Understanding belief in political statements using a model-driven experimental approach: a registered report

Abstract Misinformation harms society by affecting citizens' beliefs and behaviour. Recent research has shown that partisanship and cognitive reflection (i.e. engaging in analytical thinking) play key roles in the acceptance of misinformation. However, the relative importance of these factors remains a topic of ongoing debate. In this registered study, we tested four hypotheses on the relationship between each factor and the belief in statements made by Argentine politicians. Participants (N = 1353) classified fact-checked political statements as true or false, completed a cognitive reflection test, and reported their voting preferences. Using Signal Detection Theory and Bayesian modeling, we found a reliable positive association between political concordance and overall belief in a statement (median = 0.663, CI95 = [0.640, 0.685]), a reliable positive association between cognitive reflection and scepticism (median = 0.039, CI95 = [0.006, 0.072]), a positive but unreliable association between cognitive reflection and truth discernment (median = 0.016, CI95 = [− 0.015, 0.046]) and a negative but unreliable association between cognitive reflection and partisan bias (median = − 0.016, CI95 = [− 0.037, 0.006]). Our results highlight the need to further investigate the relationship between cognitive reflection and partisanship in different contexts and formats. Protocol registration The stage 1 protocol for this Registered Report was accepted in principle on 22 August 2022. The protocol, as accepted by the journal, can be found at: https://doi.org/10.17605/OSF.IO/EBRGC.


Parameter recovery
To demonstrate the feasibility of our methodological approach, we performed a parameter recovery analysis [Wilson2019] using fake data that we generated following the structure of the real data we will collect with the calibration and main apps (see Analysis plan).Below we describe the analysis pipeline, best understood when read alongside the corresponding R scripts in the project's OSF repository (https://osf.io/mhsr8/).
• For the calibration app analysis: 1. We generated a design matrix for 1200 subjects, including all within and between-subjects variables that will be measured.2. We specified our multivariate zero-one inflated beta generalized linear mixed model (see Analysis plan, Equation 2). 3. We fixed arbitrary -yet plausible-values for all model intercepts and for three coefficients associated with three arbitrary statements for each dependent variable.Importantly, we fixed these values by setting constant priors for each of these coefficients (henceforth, coefficients of interest).4. We sampled values for the model parameters using only the prior distributions (although for the coefficients of interest, this is actually a point value).5. We generated fake data (i.e., responses for the three dependent variables) using these sampled parameter values over the design matrix.6.We fit the fake data with the same model, although at this point -importantly-we specified weakly-informative priors for all coefficients, and used the fake data to sample the posterior distribution of the model parameters.7. We extracted the posterior distribution of each of the coefficients of interest.8.We assessed whether the fixed parameter values (used for generating fake data) were included in the 95-quantile credible interval of the posterior distribution of the corresponding coefficient of interest (Figure S4A).9. Finally, we computed the marginal (across political profiles, highest education level, and for participants' mean age) expected means for each statement for each dependent variable (ratings for pro-Macri and pro-Kirchner congruence) and we obtained the corresponding political valence values (i.e., pro-Macri congruence minus pro-Kirchner congruence rating means).These expected means were then saved to a dataframe which was later used for the parameter recovery analysis of the main app.
• For the main app analysis: 1.We generated a design matrix for 1200 subjects, including all within and between-subjects variables that will be measured (which also includes the expected political valence means for the statements, obtained from the calibration app analysis).
2. We specified our hierarchical equal-variance Signal Detection Theory (SDT) model (see Analysis plan, Equation 1). 3. We set values for the coefficients that represent hypotheses H1, H2 and H4 (see Table 1) according to the effect sizes reported in a recent review studying belief in misinformation 18 ( = 0.45, = δ _ λ _ -0.7, and = 0.2), respectively, and at a conservative "small" λ _ effect size for H3 ( = 0.2).Additionally, we fixed λ _:_ arbitrary --yet plausible--values for the model intercepts for the SDT parameters (c and d').Importantly, we fixed all these values by setting constant priors for each of these coefficients (henceforth, coefficients of interest).4. We sampled values for the model parameters using only the prior distributions (although for the coefficients of interest, this is actually a point value).5. We generated fake data (i.e., true/false responses) using these sampled parameter values over the design matrix.6.We fit the fake data with the same model, although now -importantly-we specified weakly informative priors for all coefficients, and used the fake data to sample the posterior distribution of the model parameters.7. We extracted the posterior distribution of each of the coefficients of interest.8.We assessed whether the fixed parameter values (used for generating fake data) were included in the 95-quantile credible interval of the posterior distribution of the corresponding coefficient of interest (Figure S4B).If it was included, then we interpreted this as a credible recovery of that coefficient.
Importantly, both parameter recovery analyses were successful, as evidenced by the inclusion of all the fixed parameter values within the credible intervals of the corresponding estimated coefficients' posterior distributions.This is strong support for the suitability of our analysis pipelines.We will run the same code (steps 6 and 8 for the main app analysis, and additionally step 9 for the calibration analysis) for the real collected data analysis pipeline (as detailed in the Analysis plan section).
A bat and a ball cost $1.10 in total.The bat costs $1.00 more than the ball.How much does the ball cost?
If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?
In a lake, there is a patch of lily pads.Every day, the patch doubles in size.If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?

Assessment of numeracy
Three questions in spanish and their english original version [Schwartz1997].
Imagine that we flip a fair coin 1,000 times.What is your best guess about how many times the coin would come up heads in 1,000 flips?
In the BIG BUCKS LOTTERY, the chance of winning a $10 prize is 1%.What is your best guess about how many people would win a $10 prize if 1000 people each buy a single ticket to BIG BUCKS?
In ACME PUBLISHING SWEEPSTAKES, the chance of winning a car is 1 in 1,000.

What percent of tickets to ACME PUBLISHING SWEEPSTAKES win a car?
[correct answer = 0.1 %]   Results of the parameter recovery for the main and calibration analyses (described in full in the Supplementary Information section).This procedure aims to assess whether fixed (i.e., known) model coefficients' values can be recovered by fitting our models with fake data generated with these fixed values.Fake data has the same structure as real collected data will have.A. For the calibration analysis, all coefficients were successfully recovered across dependent variables (congK (light blue), congM (light yellow) ) and distributional parameters (μ, , α and γ), as all fixed values (light red points) are ϕ contained within the 95% credible interval of the corresponding recovered posterior values (black vertical lines), in some cases very close to the median of the recovered posterior values (black dot).B. Similarly, for the main analysis, all coefficients of interest (see Table 1) were successfully recovered.Since both recovery analyses were successful, we can be confident that the proposed analytical strategy (see Analysis plan) is feasible and reliable.

Figure S1 .
Figure S1.Demographic variables of participants of the main study (N=1353)

Figure S4 .
Figure S4.Parameter recovery analysis.Results of the parameter recovery for the main and calibration analyses (described in full in the Supplementary Information section).This procedure aims to assess whether fixed (i.e., known) model coefficients' values can be recovered by fitting our models with fake data generated with these fixed values.Fake data has the same structure as real collected data will

Figure S5 .
Figure S5.Summary of estimated model coefficients.

Table S1 . Corpus of 30 statements made by political leaders and elected officials in Argentina.
Statements were previously fact-checked by Chequeado (www.chequeado.com) as either true or false.Each statement was presented to participants as detailed in the main text and Figures3 and 4. Diputado de la Nación por la Provincia de Buenos Aires): "Los intendentes de Juntos y del Frente de Todos ganan entre $ 600 mil y $ 1,2 millones".Octubre, 2021.